Assessing Agreement on Classiication Tasks: the Kappa Statistic
نویسنده
چکیده
Currently, computational linguists and cognitive scientists working in the area of discourse and dialogue argue that their subjective judgments are reliable using several diierent statistics, none of which are easily interpretable or comparable to each other. Meanwhile, researchers in content analysis have already experienced the same diiculties and come up with a solution in the kappa statistic. We discuss what is wrong with reliability measures as they are currently used for discourse and dialogue work in computational linguistics and cognitive science, and argue that we would be better oo as a eld adopting techniques from content analysis.
منابع مشابه
Assessing agreement on classification tasks: the kappa statistic
Currently, computational linguists and cognitive scientists working in the area of discourse and dialogue argue that their subjective judgments are reliable using several different statistics, none of which are easily interpretable or comparable to each other. Meanwhile, researchers in content analysis have already experienced the same difficulties and come up with a solution in the kappa stati...
متن کاملDiagnostic concordance among dermatopathologists in basal cell carcinoma subtyping: Results of a study in a skin referral hospital in Tehran, Iran
Background: Basal cell carcinomas (BCC) are the most prevalent among non-melanoma skin cancers (NMSC), which correspond to the most common skin cancers. BCC histopathological subtyping is a problem in therapeutic management. Therefore, we have decided to perform a histopathologic study for better classification of BCCs based on interobserver diagnostic judgment. Methods: We conducted this cross...
متن کاملAnalyzing Appraisal Automatically
AAAI 2004 Spring Symposium Exploring Attitude and Affect in Text Department of Linguistics Simon Fraser University Burnaby, B.C., V5A 1S6, Canada Carletta, J. 1996. Assessing agreement on classification tasks: the kappa statistic. 22 (2): 249-154. Hatzivassiloglou, V., and McKeown, K. 1997. Predicting the semantic orientation of adjectives. In , 174-181. Krippendorf, K. 1980. Beverly Hills, CA:...
متن کاملOn population-based measures of agreement
Measuring agreement between qualified experts is commonly used to determine the effectiveness of a diagnostic procedure. Many methods are available for assessing agreement, including Cohen’s kappa, which is a very popular summary measure of agreement due to its appealingly simple usage and interpretation. However, it has been previously shown that a number of flaws exist in its usage, which can...
متن کاملUnderstanding interobserver agreement: the kappa statistic.
Items such as physical exam findings, radiographic interpretations, or other diagnostic tests often rely on some degree of subjective interpretation by observers. Studies that measure the agreement between two or more observers should include a statistic that takes into account the fact that observers will sometimes agree or disagree simply by chance. The kappa statistic (or kappa coefficient) ...
متن کامل